Developing an assistive technology usability questionnaire for people with neurological diseases

Purpose This study describes the development of a questionnaire for assessing the usability of assistive technologies accessible to people with neurological diseases. Methods A Delphi study was conducted to identify relevant items for the questionnaire. After that, the content validity was addressed to identify the essential items. Once the questionnaire was designed following the results of the Delphi study and content validity, the reliability, validity, and the Rasch model of the questionnaire were examined. Results Two rounds of the Delphi study were carried out. A total of 73 participants (42 experts and 31 users) participated in round 1, and 59 people (27 experts and 32 users) in round 2. A total of 53 and 29 items were identified in rounds 1 and 2, respectively. In the content validity, we found nine items above the threshold of 0.58. Finally, ten items were included in the questionnaire. Fifty-one participants participate in the reliability and validity of the questionnaire. The internal consistency reliability of the questionnaire analyzed by Cronbach’s Alpha was α = 0,895. There was moderate to considerable concordance among our questionnaire items test-retest in the Kappa coefficient and a strong association between test-retest in the Spearman’s coefficient ρ = 0.818 (p<0,001). The intraclass correlation coefficient was 0,869 with a 95% confidence interval (0,781;0,923). There was a strong correlation between the total scores of the new questionnaire and other validated questionnaires analyzed with Spearman’s coefficient ρ = 0.756 (p<0,001). The ten items demonstrated a satisfactory fit to the Rasch model. Conclusions The present study suggested that the new questionnaire is a reliable 10-item usability questionnaire that allows subjective and quick assessment of the usability of assistive technologies by people with neurological diseases.

After that, the content validity was addressed to identify the essential items. Once the 27 questionnaire was designed following the results of Delphi study and content validity, the 28 reliability and validity, and the rasch model of the questionnaire were examined. Results. 29 Two rounds of the Delphi study were carried out. conditions. According to several authors,[33] an interval of between 2 days and 2 weeks 156 between the test-retest interviews is recommended. Accordingly, the interval between the 157 two tests was set to 15 days. In order to know the concordance between the test and the 158 retest, data was analyzed by weighted quadratic Kappa coefficient.  Table I). From round 1, 53 different items and qualitative information about 199 questionnaires were derived. 200 In round two, 59 people (27 experts and 32 users) (see table 1). From this round, 201 15 items and qualitative data about the scale were obtained. The items were: 202 "effectiveness", "comfort", "adaptability", "easy to put on/off", "safe", "lightweight", 203 "functionality", "ergonomic", "economical", "affordable", "easy to use", "feedback", 204 "stimulating," "monitored" and "movement facilitator. Due to "economical" and 205 "affordable" have the same meaning, both were considered as a single item. Additionally, 206 in this round and following the usability premises, the items "esthetics" and "easy to 207 remember how to use it" were added. Finally, 16 items were analyzed using the content 208 validity ratio. 209 210 211

**Expert in Neurorehabilitation
Between 1 and 5 years n=5 15% Between 5 and 10 years n=5 15% Between 10 and 20 years n=12 35% More than 20 years n=12 35% * Other are, for example, orthopaedic professionals and engineers. *All the participants 228 used to work with people with neurological diseases. 229 230 Participants had to choose which items were essential, useful but not essential, and 231 not essential when evaluating the items. Table 3 showed the results, nine of the sixteen 232 items exceeded the threshold of 0.58. In addition, the experts were in consensus to accept 233 the item "comfortable," because its threshold was 0.53. 234 235 0,62 *CVR (content validity ratio) = (Ne-N/2) / (N/2) with 34 people at the expert panel 237 (N=34), the items with the CVR bigger than 0.58 (in bold) will remain at the instrument 238 and the rest will be eliminated. 239

240
The experts considered "functional" and "movement facilitator" as one word due 241 to the similarity of their meanings. The item "satisfaction" was added to the questionnaire 242 to ascertain the end-user opinion of the product. Therefore, 10 items were selected to 243 create the questionnaire (see Table 4). 244 In general, _____ is easy to use Easy to use 9 Information and instructions of use______ are easy to understand and easy to remember Easy to remember how to use it 10 Overall, I am satisfied with______ Satisfaction 246 Questionnaire design 247 Following the qualitative information of the participants from the Delphi study, the 248 questionnaire was formulated in an understandable language, and the length of the 249 questionnaire was as short as possible. When necessary, the users could fill the blank 250 space with the name of the product that was being evaluated.

296
The correlation of the total scores between NATU Quest and QUEST 2.0 analyzed with 297 the Spearman's coefficient was strong with ρ = 0.756 significant, with p value < 0.0001. 298

299
All 10 items demonstrated a satisfactory fit to the Rasch model, which could be 300 considered productive for measurement (MNSQ infit between 0.64 and 1.43; MNSQ outfit 301 between 0.52 and 1.49). 302

303
There is a need for a short and easy questionaires to properly assess usability of assitive 304 techologies in people with neurological diseases. 305 The items included in the questionnaire, the format of the questionnaire and the 306 answer form were derived through two rounds of a Delphi study[25] based on the opinion 307 of 69 experts (neurorehabilitation professionals, such as medical doctors and 308 physiotherapists) and 63 users (people with neurological diseases). Finally, using a content 309 validity ratio, we narrowed items down to 10 essential usability items. Some of the items 310 also appear in other usability questionnaires such as "effectiveness" and "adaptability" are 311 included in the PIADS,[18] while "comfort," "easy to use," "safe" "easy to put on/off," 312 and "satisfaction" appears in the PIADS, Once the questionare form was designed, a total of 51 end-users with neurological 318 diseases participated in the questionnaire validity and reliablity. Results suggested that 319 NATU Quest has a good reliability and validity and it also fits in the rasch model. 320 In contrast with other questionnaires, the NATU Quest was developed taking into 321 account the professionals and people with neurological diseases' opinions. Other relevant 322 aspects of this study are the heterogeneity of the included sample, the wide range of 323 neurological diseases and the inclusion of different assistive technologies. 324 In this study, we developed a usability scale to analyze assistive technologies for 325 people with neurological diseases; however, the study had some limitations: (1) Selection 326 bias of the participants since the majority of them were from the same province. (2)  327 Although all the users who participated in the validation had a neurological disease, the 328 authors chose the Pfeiffer test to assess the cognitive problems because it is short and 329 quick to answer. However, Pfeiffer Test does not accurately assess all possible cognitive 330 deficits, and it is not sensitive enough to detect low or mild cognitive deficits. (3) Due to 331 the people with neurological diseases conditions, different assistive technologies were 332 analyzed. It would be very interesting to be able to perform another validation with the 333 same product for all users, maybe will be a good option in the developing of a new 334 product. (4) Finally, the test-retest reliability was only compared to the QUEST 2.0; 335 because we considered that adding other tests for comparison in the same study would 336 have placed a burden on the users. 337 For future work, it would be interesting to measure test-retest reliability with 338 another usability questionnaire. It would also be interesting to verify the new 339 questionnaire external validity with other population groups, such as older people. Finally, 340 the items should be reviewed after a few years to determine if they remain sufficiently 341 sensitive for assessing rapidly evolving assistive technologies. Also, should be interesting 342 to translate the new questionnaire to other languages. 343 344

345
The present study suggested that the NATU is a reliable 10-item usability questionnaire 346 that allows subjective and quick assessment of the usability of assistive technologies. This 347 questionnaire aims to be accessible to people with neurological diseases and it reflects the 348 level of acceptance and satisfaction that a patient has with the product that is using. In 349 addition, the NATU Quest can also be useful for evaluating products in development 350 through user-centered design, since the patient can state an opinion about the product 351 during its development, which will facilitate the development of products for a better fit 352 for patients' needs.